day20 : redisDB keyDB on K8S (下)

2021 iThome 鐵人賽

DAY 20

DevOps

玩耍開源k8s30天系列第 20 篇

13th鐵人賽

永和ㄦ

2021-09-20 19:08:37

2022 瀏覽

分享至

昨天簡略介紹了redis cluster的架構以及小小的講了一下keydb，所以今天會透過redis operator的方式佈署redis cluster然後用helm的方式佈署keyDB，同時會佈署兩套keyDB，一套做為獨立使用，另一套去sync redis。

curl -sL https://github.com/operator-framework/operator-lifecycle-manager/releases/download/v0.18.3/install.sh | bash -s v0.18.3

kubectl create -f https://operatorhub.io/install/redis-operator.yaml

透過這兩個指令完成redis 的operator，接著配置合適的pv 和redis-passwd 給要建立cluster 的namespaces，再配置redis cluster yaml，就可以建出redis cluster囉

apiVersion: redis.redis.opstreelabs.in/v1beta1
kind: RedisCluster
metadata:
  name: redis-cluster
spec:
  clusterSize: 3
  kubernetesConfig:
    image: 'quay.io/opstree/redis:v6.2'
    imagePullPolicy: IfNotPresent
    resources:
      requests:
        cpu: 101m
        memory: 128Mi
      limits:
        cpu: 101m
        memory: 128Mi
    redisSecret:
      name: redis-secret
      key: password
    serviceType: ClusterIP
  redisExporter:
    enabled: true
    image: 'quay.io/opstree/redis-exporter:1.0'
    imagePullPolicy: Always
    resources:
      requests:
        cpu: 100m
        memory: 128Mi
      limits:
        cpu: 100m
        memory: 128Mi
  redisLeader:
    serviceType: ClusterIP
  redisFollower:
    serviceType: ClusterIP
  storage:
    volumeClaimTemplate:
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi

另外順便提供單機版本


apiVersion: redis.redis.opstreelabs.in/v1beta1
kind: Redis
metadata:
  name: redis-standalone
spec:
  redisExporter:
    enabled: true
    image: 'quay.io/opstree/redis-exporter:1.0'
    imagePullPolicy: Always
    resources:
      requests:
        cpu: 100m
        memory: 128Mi
      limits:
        cpu: 100m
        memory: 128Mi
  kubernetesConfig:
    image: 'quay.io/opstree/redis:v6.2'
    imagePullPolicy: IfNotPresent
    resources:
      requests:
        cpu: 101m
        memory: 128Mi
      limits:
        cpu: 101m
        memory: 128Mi
    redisSecret:
      name: redis-secret
      key: password
    serviceType: LoadBalancer
  redisConfig: {}
  storage:
    volumeClaimTemplate:
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi

接著來建置一套keyDB

helm repo add enapter https://enapter.github.io/charts/
helm install my-keydb enapter/keydb --version 0.23.0 -n demo

兩套都是使用6379以及redis-cli就可以操作使用了。

接著為了實現keyDB與redis的結合，要使用keyDB中的兩個參數
active-replica yes
replicaof redis-cluster-leader 6379
第一個是主動同步
第二個則是要同步的主機IP因為使用k8s所以用service
這個要設計在keyDB的config上，所以要對sts的檔案做一些修改，

這個是會使用到的sts yaml，其中有使用一個secret檔案my-keydb-utils

spec:
  podManagementPolicy: OrderedReady
  replicas: 3
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app.kubernetes.io/instance: my-keydb
      app.kubernetes.io/name: keydb
  serviceName: my-keydb-headless
  template:
    metadata:
      annotations:
        checksum/secret-utils: df252d80f012eac3f195684b8d022babb3a58b7701a528aa8971920f9cee350a
      creationTimestamp: null
      labels:
        app.kubernetes.io/instance: my-keydb
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/name: keydb
        app.kubernetes.io/version: 6.0.18
        helm.sh/chart: keydb-0.23.0
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app.kubernetes.io/name
                  operator: In
                  values:
                  - keydb
                - key: app.kubernetes.io/instance
                  operator: In
                  values:
                  - my-keydb
              topologyKey: kubernetes.io/hostname
            weight: 100
      containers:
      - command:
        - /utils/server.sh
        image: eqalpha/keydb:x86_64_v6.0.18
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 3
          periodSeconds: 10
          successThreshold: 1
          tcpSocket:
            port: keydb
          timeoutSeconds: 1
        name: keydb
        ports:
        - containerPort: 6379
          name: keydb
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          initialDelaySeconds: 30
          periodSeconds: 10
          successThreshold: 1
          tcpSocket:
            port: keydb
          timeoutSeconds: 1
        resources: {}
        startupProbe:
          failureThreshold: 30
          periodSeconds: 5
          successThreshold: 1
          tcpSocket:
            port: keydb
          timeoutSeconds: 1
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /data
          name: keydb-data
        - mountPath: /utils
          name: utils
          readOnly: true
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
      - name: utils
        secret:
          defaultMode: 493
          items:
          - key: server.sh
            path: server.sh
          secretName: my-keydb-utils
  updateStrategy:
    rollingUpdate:
      partition: 0
    type: RollingUpdate
  volumeClaimTemplates:
  - apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      creationTimestamp: null
      name: keydb-data
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 1Gi
      volumeMode: Filesystem
    status:
      phase: Pending

apiVersion: v1
data:
  server.sh: IyEvYmluL2Jhc2gKc2V0IC1ldXhvIHBpcGVmYWlsCgpob3N0PSIkKGhvc3RuYW1lKSIKcG9ydD0iNjM3OSIKcmVwbGljYXM9KCkKZm9yIG5vZGUgaW4gezAuLjJ9OyBkbwogIGlmIFsgIiRob3N0IiAhPSAibXkta2V5ZGItJHtub2RlfSIgXTsgdGhlbgogICAgICByZXBsaWNhcys9KCItLXJlcGxpY2FvZiBteS1rZXlkYi0ke25vZGV9Lm15LWtleWRiLWhlYWRsZXNzICR7cG9ydH0iKQogIGZpCmRvbmUKZXhlYyBrZXlkYi1zZXJ2ZXIgL2V0Yy9rZXlkYi9yZWRpcy5jb25mIFwKICAgIC0tYWN0aXZlLXJlcGxpY2EgeWVzIFwKICAgIC0tbXVsdGktbWFzdGVyIHllcyBcCiAgICAtLWFwcGVuZG9ubHkgbm8gXAogICAgLS1iaW5kIDAuMC4wLjAgXAogICAgLS1wb3J0ICIkcG9ydCIgXAogICAgLS1wcm90ZWN0ZWQtbW9kZSBubyBcCiAgICAtLXNlcnZlci10aHJlYWRzIDIgXAogICAgIiR7cmVwbGljYXNbQF19Igo=

這個secret放的就是啟動keydb的shell

#!/bin/bash
set -euxo pipefail

host="$(hostname)"
port="6379"
replicas=()
for node in {0..2}; do
  if [ "$host" != "my-keydb-${node}" ]; then
      replicas+=("--replicaof my-keydb-${node}.my-keydb-headless ${port}")
  fi
done
exec keydb-server /etc/keydb/redis.conf \
    --active-replica yes \
    --multi-master yes \
    --appendonly no \
    --bind 0.0.0.0 \
    --port "$port" \
    --protected-mode no \
    --server-threads 2 \
    "${replicas[@]}"
# exit

從這段解析中發現其實已經啟動了主動sync的機制，並且設定replicaof設定為keydb自己的cluster主機群，那麼將這個目標主機指向redis就可以讓他變成redis的備份囉。